[3/5] autotune: add offline config artifacts (#770) by jhinpan · Pull Request #786 · ROCm/FlyDSL

jhinpan · 2026-07-01T07:15:47Z

Adds one optional offline-config layer to the direct @autotune flow from #785.

artifact_name= enables lookup under FLYDSL_AUTOTUNE_CONFIG_DIR.
Normal order is searched winner → matching artifact → default → search.
FLYDSL_AUTOTUNE=1 bypasses those serving decisions, searches the existing configs, updates the scratch winner, and atomically emits an artifact.
Artifact identity has one owner: the declared key, plus the call device descriptor. Adopters must put every dtype/layout/mode that changes the winner in key.
Corrupt, mismatched, or structurally invalid artifacts fall back normally. Compile or runtime errors from an accepted artifact are not masked.

RMSNorm opts in with artifact_name="rmsnorm"; its #785 direct-JIT kernel, m_in/N/dtype_str key, default, and seven BLOCK_THREADS/waves-per-EU candidates are unchanged.

There is no builder path, artifact-specific key callback, compiler/backend change, new export, or second RMSNorm factory.

Local verification on current main: 99 passed, 3 skipped across autotune, cache-key, compile-hint, external-LLVM, autotuned RMSNorm, and legacy RMSNorm tests; Python style, py_compile, diff check, and the Sphinx build pass.

Refs #770.

Copilot

Pull request overview

Adds an “offline” autotuning consumption path to FlyDSL’s autotuner so kernels can be tuned once, emit a portable JSON artifact keyed by a filename convention, and later resolve the config at runtime with zero search. This extends the existing autotune flow with builder-mode support (rebuild-per-config), a two-track default-vs-search behavior, and adopts it for RMSNorm with both unit and GPU integration tests.

Changes:

Extend flydsl.autotune.Autotuner with offline config emit/lookup, builder mode (build_fn), and a heuristic default that skips search unless FLYDSL_AUTOTUNE=1.
Add RMSNorm tuning configuration + autotuned front-end that uses builder-mode autotune and offline artifacts.
Add GPU-free unit tests and a GPU integration test covering search/no-search, caching, and offline artifact round-trip; add a user guide doc.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit/test_autotune.py	Adds GPU-free tests for cache keys, restore/reset semantics, builder mode, and offline artifact filename + round-trip.
tests/kernels/test_rmsnorm_autotune.py	Adds GPU integration coverage for RMSNorm default path, forced search + cache reuse, and offline emit/lookup.
python/flydsl/autotune.py	Implements offline config artifact support, builder-mode execution, heuristic default gating, and expanded cache key axes.
kernels/rmsnorm_kernel.py	Adds a build-time `BLOCK_THREADS` knob and conditions `known_block_size` emission for >256-thread blocks.
kernels/rmsnorm_config.py	Introduces RMSNorm default heuristic and exhaustive config set for tuning (BLOCK_THREADS × waves_per_eu).
kernels/rmsnorm_autotune.py	Adds the autotuned RMSNorm wrapper using builder mode plus offline config key extraction.
docs/autotune_guide.md	Documents the three consumption paths (default / online search / offline artifacts), cache semantics, and correctness notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Resolve merge conflicts with upstream/main (through ROCm#818) for PR ROCm#786. Conflicts, both in the shared autotune feature series (ROCm#786 is [3/5], the conflicting upstream commit is [1/5] ROCm#783 of the same series): - python/flydsl/autotune.py - tests/unit/test_autotune.py Both were resolved by taking this branch's version: PR ROCm#786 is a strict superset of upstream's autotune changes (restore_value, _run_config, _snapshot/_restore_tensors, env/toolchain/device cache-key axes) and evolves them further (builder mode, offline config emit/lookup). Taking ours also drops the auto-merge duplications (doubled snapshot line, duplicate _run_config). Verified: tests/unit/test_autotune.py passes 44/1-skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jhinpan · 2026-07-09T00:09:23Z

/rerun-ci

…fline path Blockers: - rmsnorm adopter imported kernels.rmsnorm_kernel, which no longer exists after the upstream merge relocated the kernel to kernels/norm/. Fix the imports in rmsnorm_autotune.py / rmsnorm_config.py to kernels.norm.rmsnorm_kernel; the adopter was dead-on-import (hidden by the GPU test's pre-import skip). Add a GPU-free regression test asserting the adopter's import path resolves. - Offline load accepted any JSON scalar for a config knob, so a wrong-typed value ("256" vs 256) passed validation and crashed build_fn at serve time. Type-check knob values against the heuristic default's config, and wrap the offline-sourced run so a bad artifact falls back to default/search instead of crashing the serving call. Robustness / serving hot path: - Negative-cache the resolved decision (default/offline) so a missing or rejected artifact isn't re-stat/parsed and re-warned on every serving call. - Route serving-path warnings through logging (warn-once per artifact) instead of unconditional print to stdout; keep tuning progress on stdout. - Atomic writes (tempfile + os.replace) for offline artifacts and the disk cache, so a concurrent reader never sees a torn file. - Persist only searched bests to the disk cache (not memoized default/offline configs), so they can't shadow a committed artifact next process. - Serialize the compiler-hint mutate/restore on the shared launch fn with a lock (thread-safe serving of hinted configs). - Reject non-finite float knob values; sanitize the scratch-cache filename; guard offline path ops against OSError (e.g. ENAMETOOLONG); warn on filename sanitization collisions at emit; warn when builder mode drops non-structural config kwargs; invalidate memoized offline decisions if CONFIG_DIR changes; gate restore_value snapshot on copy_ support. rmsnorm: - get_all_configs no longer emits a single-config "search" for f32; it returns the heuristic default explicitly (the BLOCK_THREADS sweep is 16-bit only) and reuses the kernel's VEC_WIDTH as the single source of truth for tile width. GPU-free unit tests extended (48 total): mistyped-scalar rejection, no-reread/ no-respam on rejected artifacts, and the adopter import-path guard. ruff + black clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

jhinpan · 2026-07-09T01:07:06Z

/rerun-ci

jhinpan · 2026-07-13T20:55:05Z

/rerun-ci

jhinpan

Two non-blocking environment-configuration nits; neither is a correctness issue.

Co-authored-by: Cursor <cursoragent@cursor.com>

zhiding512 · 2026-07-22T06:30:04Z

+    return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False)
+
+
+def _atomic_write_text(path: Path, text: str) -> None:


There is also an atomic write function in JitCacheManager (jit_function.py). Can we improve it ? Reuse this function ?

Fixed in ecf4fe7. I extracted the same-directory temp-file + atomic replacement protocol into flydsl.utils.file.atomic_write and reused it from both JitCacheManager (binary pickle) and autotune artifacts (UTF-8 JSON). Added coverage for replacement visibility, failure cleanup, and binary writes.

Co-authored-by: Cursor <cursoragent@cursor.com>

jhinpan · 2026-07-23T07:14:53Z

/rerun-ci

Copilot AI review requested due to automatic review settings July 1, 2026 07:15

Copilot started reviewing on behalf of jhinpan July 1, 2026 07:16 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread python/flydsl/autotune.py Outdated

Comment thread python/flydsl/autotune.py Outdated

Comment thread python/flydsl/autotune.py

Comment thread kernels/rmsnorm_autotune.py Outdated

jhinpan force-pushed the feat/autotune-offline-configs branch from bd834da to b7b3c25 Compare July 1, 2026 08:02

jhinpan mentioned this pull request Jul 1, 2026

autotune: keep CI search opt-in (#770) #788

Open

jhinpan force-pushed the feat/autotune-offline-configs branch 11 times, most recently from 5e2cda0 to e7edf3b Compare July 2, 2026 19:27

jhinpan mentioned this pull request Jul 9, 2026

[Feature] Make autotune a safe, adopted path #770

Open

6 tasks

jhinpan mentioned this pull request Jul 13, 2026

[CI] run_benchmark: treat runtime pytest.skip as skipped, not failed #842

Merged

3 tasks

jhinpan force-pushed the feat/autotune-offline-configs branch from 1da82fc to 1633bf7 Compare July 18, 2026 03:43

jhinpan changed the title ~~[3/5] autotune: offline config emit + runtime lookup (#770)~~ [3/5] autotune: add opt-in offline config artifacts (#770) Jul 18, 2026

jhinpan force-pushed the feat/autotune-offline-configs branch from 1633bf7 to ff067a9 Compare July 18, 2026 04:33

jhinpan changed the title ~~[3/5] autotune: add opt-in offline config artifacts (#770)~~ [3/5] autotune: add offline config artifacts (#770) Jul 18, 2026

autotune: add opt-in offline config artifacts

09dd110

jhinpan force-pushed the feat/autotune-offline-configs branch from ff067a9 to 09dd110 Compare July 20, 2026 07:56

Merge branch 'main' into feat/autotune-offline-configs

b4f96b4

jhinpan commented Jul 21, 2026

View reviewed changes

Comment thread python/flydsl/autotune.py Outdated

Comment thread python/flydsl/autotune.py Outdated

refactor(autotune): centralize artifact config policy

18dfd2b

Co-authored-by: Cursor <cursoragent@cursor.com>

jhinpan mentioned this pull request Jul 22, 2026

[RFC] FlyDSL kernels in downstream libraries #749

Open

zhiding512 reviewed Jul 22, 2026

View reviewed changes

jhinpan and others added 3 commits July 22, 2026 20:15

refactor(utils): centralize atomic file writes

ecf4fe7

Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'main' into feat/autotune-offline-configs

cc9e1df

fix(autotune): harden offline config caching

ba7e140

Co-authored-by: Cursor <cursoragent@cursor.com>

jhinpan mentioned this pull request Jul 22, 2026

[rmsnorm] Support FP32 weights with FP16/BF16 activations #884

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[3/5] autotune: add offline config artifacts (#770)#786

[3/5] autotune: add offline config artifacts (#770)#786
jhinpan wants to merge 6 commits into
ROCm:mainfrom
jhinpan:feat/autotune-offline-configs

jhinpan commented Jul 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhinpan commented Jul 9, 2026

Uh oh!

jhinpan commented Jul 9, 2026

Uh oh!

jhinpan commented Jul 13, 2026

Uh oh!

jhinpan left a comment

Uh oh!

Uh oh!

Uh oh!

zhiding512 Jul 22, 2026

Uh oh!

jhinpan Jul 22, 2026

Uh oh!

jhinpan commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False)


		def _atomic_write_text(path: Path, text: str) -> None:

Uh oh!

Conversation

jhinpan commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhinpan commented Jul 9, 2026

Uh oh!

jhinpan commented Jul 9, 2026

Uh oh!

jhinpan commented Jul 13, 2026

Uh oh!

jhinpan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhiding512 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

jhinpan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

jhinpan commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jhinpan commented Jul 1, 2026 •

edited

Loading